Hidden Markov model multiarm bandits: a methodology for beam scheduling in multitarget tracking

In this paper, we derive optimal and suboptimal beam scheduling algorithms for electronically scanned array tracking systems. We formulate the scheduling problem as a multiarm bandit problem involving hidden Markov models (HMMs). A finite-dimensional optimal solution to this multiarm bandit problem is presented. The key to solving any multiarm bandit problem is to compute the Gittins index. We present a finite-dimensional algorithm that computes the Gittins index. Suboptimal algorithms for computing the Gittins index are also presented. Numerical examples are presented to illustrate the algorithms.